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Abstract 


~We-provei>^  classical  limit  theorem  on  characteristic  functions 
by  using  duality  between  a  pair  of  optimization  problems,  one  of 
which  is  w  infinite  dimensional  minimization  involving  the  relative 
entropy  functional. 
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Introduction 


Modern  optimization  theory  has  been  employed  successfully  in  many 
diverse  fields  such  as  Economics,  Physics,  Statistics,  Biology  and 
Engineering.  This  paper  is  a  small  step  toward  demonstrating  the  use 
of  optimization  theory  as  proof  mechanism  in  Probability. 

The  result  from  Probability  Theory  in  question  here  is  a  limit 
theorem  on  characteristic  functions.  Let  X  be  a  random  vauriable  with 

distribution  F  ,  support  Ix_ ,x  ]  and  characteristic  function 

-  .  itX 

♦  ct)  -  Ee 

Then 

X  ■  lim  —  log  i|i(-iy) 

*  y 

X  ■  -lim  i  log  t|)(iy)  . 

"  y-*<«  y 

The  result  Is  given  in  Lucas'  classical  book  "Characteristic  Functions" 

[3];  first  a  weaker  result,  concerning  only  analytic  characteristic  functions, 
is  proved  in  Chapter  7.  The  full  statement  is  given  in  Chapter  11,  as  part 
of  Theorem  11.1.2.  It  is  derived  from  the  result  in  Ch.  7  via  a  chain  of 
lemmas  on  boundary  characttetistic  functions. 

Here  we  prove  the  above  limit  theorem  by  using  duality  relations 
between  two  extremum  problems.  One  of  these  problems  is  em  infinite¬ 
dimensional  convex  program  involving  the  minimization  of  the  relative 
entropy  functional,  which  is  of  dundamental  importance  in  Statistical 
Information  Theory,  Thermodynamics  and  Communication  Theory. 

The  plan  of  the  paper  is  as  follows:  Section  1  gives  a  formal 
statement  of  the  limit  theorem  (Theorem  A) .  Section  2  gives  the  duality 
theorem  (Theorem  B) ,  vAiich  is  in  fact  an  adaptation  of  a  result  in  the 
authors*  paper  IIJ .  In  Section  3  we  prove  Theorem  A  via  Theorem  B. 


A  limit  theorem  on  characteristic  functions 


Let  X  be  a  random  variable,  and  F  (x)  its  distribution  function. 

A 

Let  i|)(t)  denote  the  characteristic  function  of  F  ,  i.e. 

A 

00 

I|){t)  =  /  e^^*  dF^(x)  . 

»00 

The  left  extremity  of  F^  is  the  number  x^^  with  the  property 
Ve  >  0  :  F  (x  -e)  =  0  ,  F^{x_+e)  >  0  . 

A  Li  A  li 

and  the  right  extremity  of  F  is  the  number  X^^  with  the  property 
Vc  >  0  ;  ^  ^ 

The  interval  support  of  F^^.  Clearly 

""R  itx 

ij.(t)  =  /  e*^  dFj^(x)  . 

*L 

Pjj  is  bounded  to  the  left  if  >-•  and  bounded  to  the  right  if 
*r"  "  • 

Theorem  A.  If  be  bounded  to  the  right,  then  its  right  extremity 

i»  given  by 

(1)  Xj^  -  lim  ^  log  i|;(-iy) 

If  F  is  bounded  from  the  left,  then  Its  left  extremity  is  given  by 

(2)  X  «  -lim  ^  log  iji  (iyj  . 

^  y 


A  duality  theorem  on  relative  entropies 

dF 

Let  D  be  the  class  of  generalized  densities  f  “  -rr  (Radon-Ni kQdym 

dt 

derivatives)  of  distribution  functions  F,  on  a  given  probability 


space,  with  support  [x. ,x_l 

L  R 


In  particular  F^  C  D  and  f^ 


is  the  corresponding  density.  The  relative  entropy  (divergence , 
Kullback-Leibler  distance)  between  P  €  D  and  is  given  by  the 

quantity 

-  /  f  (t)logl|^j]dt  . 

X# 

It  is  well  known  that  is  a  nonnegative  convex  functional 

amd  is  equal  to  zero  if  and  only  if  f  >■  f^  (a.e.  with  respect  to  dt) 
see  [2] . 

A  special  case  of  a  problem  studied  in  [l,Ch.3]  is  the  infinite-dimensional 
convex  program 

(E)  inf{l(f,f  )  I  /  g(t)f(t)dt  >  a} 

«eD  *  xj. 

where  g(t)  is  a  given  susinable  function.  It  was  shown  in  [I]  that  a  dual 
problem  is  given  by 

(H)  sup{ay-log  /  f  Ct)dt}  , 

y>0  * 

Moreover,  from  Th.  1  in  11]  tWh  following  duality  relations  hold 
between  (E)  and  (H) . 

Theorem  B.  If  (E)  is  feasible  then  inf (E)  is  attained,  8up(H)  is 
finite  and 

min(E)  «  sup(H)  . 

Proof  of  Theorem  A  via  Theorem  B 


First  note  that  the  trivial  inequality 


implies 


lim  —  log  E  e 
y 

y»«»  ■* 


yX  <  X 


R 


i.e. 

(3)  lim  —  log  ij;(-iy)  <  x  . 

y*H<i  y  “  * 


Consider  now  the  problem 


*R 

(E^)  inf{I(f;fJ:  I  tf(t)dt  >  -  e} 

®  £60  ■  X,  “  R 


for  some  fixed  e  >  0  .  This  is  a  special  case  of  problem  (E)  with 
g(t)  a  t  ,  a  •  X  •  e  .  The  dual  is 

Jl\ 

*R  vt 

8up{y(x  -c)  -  log  /  e^  f  it)dt> 

i.e. 

(D,)  8up{y(x  -€)  -  log  \(i(-iy)} 

y>0  ^ 


Problem  (E  )  is  clearly  feasible  for  every  e  >  0  ,  and  we  infer  from 
e 

Theorem  B: 


•  >  sgp(D  )  >  lim{y(x  -e)  -  log  ji(-iy)}  ■ 

®  ”  y-H»  * 

■  lim  y  IXj^-e  -  ^  log  (|»  (-iy)  J 

y+oD  ^ 

Now I  for  the  limit  to  be  finite,  it  is  necessary  that: 

(4)  X  -  €  <  lim  —  log  ij»(-iy)  y  e  >  0  . 

*  "  ^ 

Conibining  (3)  and  (4)  we  obtain  equation  (1) . 


To  prove  equation  (2)  we  note  that  the  inequality 


(5)  -lim  i  log  i(»  (iy)  > 

y*«>  ^ 

is  trivial,  while  the  Inequality 

(6)  -lim  —  log  4»(iy)  <x  +e  Ve>0 

y-xo  y  “  ^ 

follows  by  applying  Theorem  B  (in  the  above  memner)  to  the  dual  pair 

inf{l(f,f  )  j  /  tf(t)dt  >  X  -  e} 

*  X  ‘  ^ 

I. 

sup{-y(x  +e)  -  log  <|)(iy)}  . 
y>0  ^ 

Now,  (5)  and  (6)  indeed  imply  (2)  ,  and  the  proof  of  Theorem  A  is 
thereby  ccsnpleted. 
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